Skip to content

Appends the saved objects documents count to the CoreUsageData service#124308

Merged
TinaHeiligers merged 6 commits intoelastic:mainfrom
TinaHeiligers:core-usage-data-use-count-api
Feb 3, 2022
Merged

Appends the saved objects documents count to the CoreUsageData service#124308
TinaHeiligers merged 6 commits intoelastic:mainfrom
TinaHeiligers:core-usage-data-use-count-api

Conversation

@TinaHeiligers
Copy link
Copy Markdown
Contributor

@TinaHeiligers TinaHeiligers commented Feb 2, 2022

Resolves #120485

This PR adds the actually saved objects document count to the CoreUsageDataService payload, retaining the original Lucene document count and other information about the store size from the GET _cat/indices/.kibana API.
In addition, this PR adds debug level logs for the Lucene document count and the saved objects documents count to make debugging easier.

Why not just change what we report?

I decided to keep the original payload and add the additional information rather than replace the docs count for 2 reasons:

  1. Reindexing the upstream data is a pain but would need to be done if we were to rename the fields or replace them.
  2. We already have data reporting the Lucene document count and other useful information (primary store size) from the _cat/indices/.kibana API. It would be confusing to simply replace the lucene docs count with that of the saved objects docs count because that's not what the _cat/indices/.kibana API API returns.

Changes:

In the example data payload (Advanced settings -> Usage Data -> cluster data:

Screen Shot 2022-02-02 at 11 56 28

In the logs

Logging configuration

// kibana.yml
appenders:
    console:
      type: console
      layout:
        type: pattern
        highlight: true
        pattern: "[%date][%level][%logger] [%meta]---%message"
root:
    appenders: [default, console]
    level: warn
  loggers:
    - name: core-usage-stats-service
      appenders: [console]
      level: debug

Logs

2022-02-02T11:56:08.425-07:00][DEBUG][core-usage-stats-service] []---Lucene documents count 267 from index .kibana
[2022-02-02T11:56:08.426-07:00][DEBUG][core-usage-stats-service] []---Saved objects documents count 148 from index .kibana
[2022-02-02T11:56:08.448-07:00][DEBUG][core-usage-stats-service] []---Lucene documents count 18 from index .kibana_task_manager
[2022-02-02T11:56:08.448-07:00][DEBUG][core-usage-stats-service] []---Saved objects documents count 18 from index .kibana_task_manager

How to test this:

  • Configure logging for the core-usage-stats-service in kibana.yml as described above
  • Start elasticsearch and kibana
  • Navigate to Advanced Settings -> Usage Data and click on cluster stats to see an example of the data Kibana collects
  • Search for savedObjects under services in the flyout and observe that we report the savedObjectsDocsCount as another entry per index.
  • Logs: search for the core-usage-stats-service logs and observe we have two entries for docs counts: one for the Lucene document count and one for the saved objects document counts.

Checklist

Risk Matrix

Risk Probability Severity Mitigation/Notes
Making an additional API call reduces telemetry collection performance Low Low Caching and the backoff mechanism will ensure that usage data collection does not impact cluster performance to any large degree.

For maintainers

…e as savedObjectsDocsCounts to distinguish it from the lucene docs count we already report
@TinaHeiligers
Copy link
Copy Markdown
Contributor Author

@elasticmachine merge upstream

_meta: {
description:
'The number of documents in the index, including hidden nested documents.',
'The number of lucene documents in the index, including hidden nested documents.',
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updates the description to make it clear what the number of documents is from.

Copy link
Copy Markdown
Contributor Author

@TinaHeiligers TinaHeiligers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self review

@kibana-ci
Copy link
Copy Markdown

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@TinaHeiligers TinaHeiligers marked this pull request as ready for review February 2, 2022 19:37
@TinaHeiligers TinaHeiligers requested review from a team as code owners February 2, 2022 19:37
@TinaHeiligers TinaHeiligers requested a review from rudolf February 3, 2022 06:29
@rudolf rudolf added the bug Fixes for quality problems that affect the customer experience label Feb 3, 2022
@rudolf
Copy link
Copy Markdown
Contributor

rudolf commented Feb 3, 2022

I agree with using a new field name, but I wonder if we have no use cases to use the lucene document/deleted count, isn't it better to stop sending it? Although it feels fairly cheap to send two more fields, everything adds up in the snapshot size and telemetry cluster and I can't imagine the lucene document count ever being useful.

@TinaHeiligers
Copy link
Copy Markdown
Contributor Author

TinaHeiligers commented Feb 3, 2022

... everything adds up in the snapshot size and telemetry cluster...

Agreed, however, I was also thinking about getting the so counts sooner rather than later, and removing fields means a reindex has to happen. Reindexing data has in my experience been a pain.

@TinaHeiligers TinaHeiligers merged commit ad7c8de into elastic:main Feb 3, 2022
@TinaHeiligers TinaHeiligers deleted the core-usage-data-use-count-api branch February 3, 2022 14:56
@TinaHeiligers TinaHeiligers added the auto-backport Deprecated - use backport:version if exact versions are needed label Feb 3, 2022
@kibanamachine
Copy link
Copy Markdown
Contributor

The following labels were identified as gaps in your version labels and will be added automatically:

  • v8.2.0

If any of these should not be on your pull request, please manually remove them.

kibanamachine added a commit to kibanamachine/kibana that referenced this pull request Feb 3, 2022
elastic#124308)

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit ad7c8de)
@kibanamachine
Copy link
Copy Markdown
Contributor

💔 Some backports could not be created

Status Branch Result
8.0 Backport failed because of merge conflicts
8.1
7.17 Backport failed because of merge conflicts

How to fix

Re-run the backport manually:

node scripts/backport --pr 124308

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Feb 3, 2022
#124308) (#124536)

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit ad7c8de)

Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
TinaHeiligers added a commit to TinaHeiligers/kibana that referenced this pull request Feb 3, 2022
elastic#124308)

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit ad7c8de)

# Conflicts:
#	docs/development/core/server/kibana-plugin-core-server.md
#	docs/development/core/server/kibana-plugin-core-server.mergesavedobjectmigrationmaps.md
TinaHeiligers added a commit to TinaHeiligers/kibana that referenced this pull request Feb 3, 2022
elastic#124308)

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit ad7c8de)

# Conflicts:
#	docs/development/core/server/kibana-plugin-core-server.md
#	docs/development/core/server/kibana-plugin-core-server.mergesavedobjectmigrationmaps.md
TinaHeiligers added a commit that referenced this pull request Feb 3, 2022
#124308) (#124562)

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit ad7c8de)

# Conflicts:
#	docs/development/core/server/kibana-plugin-core-server.md
#	docs/development/core/server/kibana-plugin-core-server.mergesavedobjectmigrationmaps.md
TinaHeiligers added a commit that referenced this pull request Feb 3, 2022
#124308) (#124563)

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit ad7c8de)

# Conflicts:
#	docs/development/core/server/kibana-plugin-core-server.md
#	docs/development/core/server/kibana-plugin-core-server.mergesavedobjectmigrationmaps.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-backport Deprecated - use backport:version if exact versions are needed bug Fixes for quality problems that affect the customer experience Feature:Saved Objects release_note:skip Skip the PR/issue when compiling release notes v7.17.1 v8.0.0 v8.1.0 v8.2.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CoreUsageDataService savedObjects.indices[0].docsCount should use _count API

5 participants